Method of notifying users regarding motion artifacts based on image analysis

ABSTRACT

A digital image acquisition system includes a portable apparatus for capturing digital images and a digital processing component for detecting, analyzing and informing the photographer regarding motion blur, and for reducing camera motion blur in an image captured by the apparatus. The digital processing component operates by comparing the image with at least one other image, for example a preview image, of nominally the same scene taken outside the exposure period of the main image. In one embodiment the digital processing component determines the degree of artefacts and whether to inform the user that the image is blurred by identifying at least one feature in a single preview image which is relatively less blurred than the corresponding feature in the main image. In another embodiment, the digital processing component calculates a trajectory of at least one feature in a plurality of preview images, extrapolates such feature on to the main image, calculates a PSF in respect of the feature, and informs the user based on the calculated PSF. In another embodiment the digital processing unit after determining the degree of blur notifies the photographer of the existing blur or automatically invokes consecutive captures. In another embodiment, the digital processing unit determines whether the image quality is acceptable from real time analysis of the captured image and provides this information to the user. Such real time analysis may use the auto focusing mechanism to qualitatively determine the PSF.

FIELD OF THE INVENTION

This invention relates to a digital image acquisition system comprisinga digital processing component for determining motion blurringartifacts, and preferably a camera motion blur function, in a captureddigital image.

BACKGROUND TO THE INVENTION

Camera motion is dependent on a few parameters. First of all, theexposure speed. The longer the shutter is open, the more likely thatmovement will be noticed. The second is the focal length of the camera.The longer the lens is, the more noticeable the movement is. A rule ofthumb for amateur photographers shooting 35 mm film is never to exceedthe exposure time beyond the focal length, so that for a 30 mm lens, notto shoot slower than 1/30th of a second.

The third criteria is the subject itself. Flat areas, or low frequencydata, is less likely to be degraded as much as high frequency data.

Historically, the problem was addressed by anchoring the camera, such aswith the use of a tripod or monopod, or stabilizing it such as with theuse of gyroscopic stabilizers in the lens or camera body, or movement ofthe sensor plane to counteract the camera movement.

Mathematically, the motion blurring can be explained as applying a PointSpread Function, or PSF, to each point in the object. This PSF representthe path of the camera, during the exposure integration time. Motion PSFis a function of the motion path and the motion speed, which determinesthe integration time, or the accumulated energy for each point.

A hypothetical example of such a PSF is illustrated in FIGS. 3-a and3-b. FIG. 3-b is a projection of FIG. 3-a. In FIGS. 3-a and 3-b, the PSFis depicted by 410 and 442 respectively. The pixel displacement in x andy directions are depicted by blocks 420 and 421 respectively for the Xaxis and 430 and 432 for the Y axis respectively. The energy 440 is thethird dimension of FIG. 3-a. Note that the energy is the inverse of thedifferential speed in each point, or directly proportional to the timein each point. In other words, the longer the camera is stationary at agiven location, the longer the integration time is, and thus the higherthe energy packed. This may also be depicted as the width of the curve442 in a X-Y projection.

Visually, when referring to images, in a simplified manner, FIG. 3-cillustrates what would happen to a pinpoint white point in an imageblurred by the PSF of the aforementioned Figures. In a picture, suchpoint of light surrounded by black background will result in an imagesimilar to the one of FIG. 3-c. In such image, the regions that thecamera was stationary longer, such as 444 will be brighter than theregion where the camera was stationary only a fraction of that time.Thus such image may provide a visual speedometer, or visualaccelerometer. Moreover, in a synthetic photographic environment suchknowledge of a single point, also referred to as a delta-function coulddefine the PSF.

Given:

-   -   a two dimensional image I represented by I(x,y)    -   a motion point spread function MPSF(I)    -   The degraded image I′(x,y) can be mathematically defined as the        convolution of I(X,Y) and MPSF(x,y) or        I′(x,y)=I(x,y){circle around (×)}MPSF(x,y)   (Eq. 1)

or in the integral form for a continuous functionI(x,y)=∫∫(I(x−x′,y−y′)MPSF(x′y′)∂x′∂y′  (Eq. 2)

and for a discrete function such as digitized images: $\begin{matrix}{{I^{\prime}( {m,n} )} = {\sum\limits_{j}{\sum\limits_{k}{{I( {{m - j},{n - k}} )}{{MPSF}( {j,k} )}}}}} & ( {{Eq}.\quad 3} )\end{matrix}$

Another well known PSF in photography and in optics in general isblurring created by de-focusing. The different is that de-focusing canusually be depicted by primarily a symmetrical Gaussian shift invariantPSF, while motion de-blurring is not. In addition, focus is a localattributes meaning some regions of the image may be in focus whileothers are not, while motion affects the entire image, even if not in anequal, shift invariant fashion. However, in many cases, the qualitativenotion of whether the image was blurred by lack of focus or motion maybe similar in nature.

The reason why motion de-blurring is not shift invariant is that theimage may not only shift but also rotate. Therefore, a completedescription of the motion blurring is an Affine transform that combinesshift and rotation based on the following transformation:$\begin{matrix}{\begin{bmatrix}u \\v \\1\end{bmatrix} = \begin{bmatrix}{{Cos}\quad\omega} & {{Sin}\quad\omega} & {\Delta\quad x} \\{{- {Sin}}\quad\omega} & {\cos\quad\omega} & {\Delta\quad y} \\0 & 0 & 1\end{bmatrix}} & ( {{Eq}.\quad 4} )\end{matrix}$

The PSF can be obtained empirically as part of a more generic field suchas system identification. For linear systems, the PSF can be determinedby obtaining the system's response to a known input and then solving theassociated inversion problems.

The known input can be for an optical system, a point, alsomathematically defined in the continuous world as a delta function δ(x),a line, an edge or a corner.

An example of a PSF can be found in many text books such as“Deconvolution of Images and Spectra” 2nd. Edition, Academic Press,1997, edited by Jannson, Peter A. and “Digital Image Restoration”,Prentice Hall, 1977 authored by Andrews, H. C. and Hunt, B. R.

The process of de-blurring an image is done using de-convolution whichis the mathematical form of separating between the convolve image andthe convolution kernel. However, as discussed in many publications suchas Chapter 1 of “Deconvolution of Images and Spectra” 2nd. Edition,Academic Press, 1997, edited by Jannson, Peter A., the problem ofde-convolution can be either unsolvable, ill-posed or ill-conditioned.Moreover, for a physical real life system, an attempt to find a solutionmay also be exacerbated in the presence of noise or sampling.

One may mathematically try and perform the restoration viade-convolution means without the knowledge of the kernel or in this casethe PSF. Such methods known also as blind de-convolution. The results ofsuch process with no a-priori knowledge of the PSF for a general opticalsystem are far from acceptable and require extensive computation.Solutions based on blind de-convolution may be found for specificcircumstances as described in “Automatic multidimensional deconvolution”J. Opt. Soc. Am. A, vol. 4(1), pp. 180-188, January 1987 to Lane et al,“Some Implications of Zero Sheets for Blind Deconvolution and PhaseRetrieval”, J. Optical Soc. Am. A, vol. 7, pp. 468-479, 1990 to Bates etal, Iterative blind deconvolution algorithm applied to phase retrieval”,J. Opt. Soc. Am. A, vol. 7(3), pp. 428-433, March 1990 to Seldin et aland “Deconvolution and Phase Retrieval With Use of Zero Sheets,” J.Optical Soc. Am. A, vol. 12, pp. 1,842-1,857, 1995 to Bones et al.However, as known to those familiar in the art of image restoration, andas explained in “Digital Image Restoration”, Prentice Hall, 1977authored by Andrews, H. C. and Hunt, B. R., blurred images can besubstantially better restored when the blur function is known.

The article “Motion Deblurring Using Hybrid Imaging”, by Moshe Ben-Ezraand Shree K. Nayar, from the Proceedings IEEE Computer SocietyConference on Computer Vision and Pattern Recognition, 2003, determinesthe PSF of a blurred image by using a hybrid camera which takes a numberof relatively sharp reference images during the exposure period of themain image. However, this requires a special construction of camera andalso requires simultaneous capture of images. Thus this technique is notreadily transferable to cheap, mass-market digital cameras.

It is an object of the invention to provide an improved technique fordetermining a camera motion blur function in a captured digital imagewhich can take advantage of existing camera functionality and does nottherefore require special measurement hardware (although the use of theinvention in special or non-standard cameras is not ruled out).

SUMMARY OF THE INVENTION

According to the present invention there is provided a digital imageacquisition system comprising an apparatus for capturing digital imagesand a digital processing component for warning a photographerdetermining that a threshold camera motion blur has occurred in acaptured digital image. The determination is based on a comparison of atleast two sets of image data each acquired within a temporal range thatincludes an exposure period of the captured digital image, and timesproximately before and after said exposure period, and of nominally thesame scene as that of the captured digital image.

Preferably, the at least two images comprise the captured image andanother image taken outside, preferably before and alternatively after,the exposure period of said captured image.

Preferably at least one reference image is a preview image.

Preferably, too, said digital image acquisition system is a portabledigital camera.

In one embodiment the digital processing component identifies at leastone characteristic in a single reference image which is relatively lessblurred than the corresponding feature in the captured image, andcalculates a point spread function (PSF) in respect of saidcharacteristic.

A characteristic as used in this invention may be a well-definedpattern. The better the pattern is differentiated from its surroundings,such as by local contrast gradient, local color gradient, well-definededges, etc., the better such pattern can be used to calculate the PSF.In an extreme case, the pattern forming the characteristic can be only asingle pixel in size.

In another embodiment the digital processing component calculates atrajectory of at least one characteristic in a plurality of referenceimages, extrapolates such characteristic on to the captured image, andcalculates a PSF in respect of said characteristic.

In either case, based on the calculated PSF, the captured image can bedeblurred using a de-convolution technique.

In yet another embodiment, the digital processing component analyses theimage motion blur in real time based on the captured image and providesa notification to the user when determined that the acquired image isnot of acceptable quality due to motion blur.

Corresponding de-blurring function determining methods are alsoprovided. One or more storage devices are also provided having digitalcode embedded thereon for programming one or more processors to performthe de-blurring function determining methods.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of example,with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of a camera apparatus operating in accordancewith an embodiment of the present invention.

FIG. 2 illustrates the workflow of the initial stage of a camera motionblur reducing means using preview data according to embodiments of theinvention.

FIGS. 3-a to 3-c illustrate an example of a point spread function (PSF).

FIG. 4 is a workflow illustrating a first embodiment of the invention.

FIG. 5 is a workflow illustrating a second embodiment of the invention.

FIGS. 6 and 7-a and 7-b are diagrams which assist in the understandingof the second embodiment.

DESCRIPTION OF A PREFERRED EMBODIMENT

FIG. 1 shows a block diagram of an image acquisition system such as adigital camera apparatus operating in accordance with the presentinvention. The digital acquisition device, in this case a portabledigital camera 20, includes a processor 120. It can be appreciated thatmany of the processes implemented in the digital camera may beimplemented in or controlled by software operating in a microprocessor(μProc), central processing unit (CPU), controller, digital signalprocessor (DSP) and/or an application specific integrated circuit(ASIC), collectively depicted as block 120 and termed as “processor”.Generically, all user interface and control of peripheral componentssuch as buttons and display is controlled by a μ-controller 122.

The processor 120, in response to a user input at 122, such as halfpressing a shutter button (pre-capture mode 32 ), initiates and controlsthe digital photographic process. Ambient light exposure is determinedusing light sensor 40 in order to automatically determine if a flash isto be used. The distance to the subject is determined using focusingmeans 50 which also focuses the image on image capture means 60. Thefocusing means may also involve a image processing mechanism to detectblurred image. This mechanism may be used to detect not only blurredimages due to de-focusing but also blurred image due to motionartefacts. If a flash is to be used, processor 120 causes the flashmeans 70 to generate a photographic flash in substantial coincidencewith the recording of the image by image capture means 60 upon fulldepression of the shutter button. The image capture means 60 digitallyrecords the image in colour. The image capture means is known to thosefamiliar with the art and may include a CCD (charge coupled device) orCMOS to facilitate digital recording. The flash may be selectivelygenerated either in response to the light sensor 40 or a manual input 72from the user of the camera.

The image recorded by image capture means 60 is stored in image storemeans 80 which may comprise computer memory such a dynamic random accessmemory or a non-volatile memory. The camera is equipped with a display100, such as an LCD at the back of the camera or a microdisplay insidethe viewfinder, for preview and post-view of images. In the case ofpreview images, which are generated in the pre-capture mode 32, thedisplay 100 can assist the user in composing the image, as well as beingused to determine focusing and exposure. A temporary storage space 82 isused to store one or plurality of the preview images and be part of theimage store means 80 or a separate component. The preview image isusually generated by the same image capture means 60, and for speed andmemory efficiency reasons may be generated by subsampling the image 124using software which can be part of the general processor 120 ordedicated hardware, before displaying 100 or storing 82 the previewimage.

Upon full depression of the shutter button, a full resolution image isacquired and stored, 80. The image may go through image processingstages such as conversion from the RAW sensor pattern to RGB, format,color correction and image enhancements. These operations may beperformed as part of the main processor 120 or by using a secondaryprocessor such as a dedicated DSP. Upon completion of the imageprocessing the images are stored in a long term persistent storage suchas a removable storage device 112.

According to this embodiment, the system includes a motion de-blurringcomponent 100. This component can be implemented as firmware or softwarerunning on the main processor 120 or on a separate processor.Alternatively, this component may be implemented in software running onan external processing device 10, such as a desktop or a server, whichreceives the images from the camera storage 112 via the image outputmechanism 110, which can be physical removable storage, wireless ortethered connection between the camera and the external device. Themotion de-blurring component 100 includes a PSF calculator 110 and animage de-convolver 130 which de-convolves the full resolution imageusing the PSF. These two components may be combined or treatedseparately. The PSF calculator 110 may be used for qualification only,such as determining if motion blur exists, while the image de-convolver130 may be activated only after the PSF calculator 110 has determined ifde-blurring is needed.

FIG. 2 is a flow chart of one embodiment of calculating the PSF inaccordance with the present invention. While the camera is in previewmode, 210, the camera continuously acquires preview images, calculatingexposure and focus and displaying the composition. When such an imagesatisfies some predefined criteria 222, the preview image is saved, 230.As explained below, such criteria will be defined based on image qualityand/or chronological considerations. A simple criteria may be alwayssave the last image. More advanced image quality criteria may includeanalysis as to whether the preview image itself has too much motionblurring. As an alternative to saving a single image, multiple imagesmay be saved, 240, the newest preview image being added to the list,replacing the oldest one, 242 and 244. The definition of oldest can bechronological, as in First In First Out. Alternatively it can be theimage that least satisfies criteria as defined in stage 222. The processcontinues, 211, until the shutter release is fully pressed, 280, or thecamera is turned off.

The criteria, 222, that a preview image needs to satisfy can varydepending on specific implementations of the algorithm. In one preferredembodiment, such criteria may be whether the image is not blurred. Thisis based on the assumption that even if a camera is constantly moving,being hand held by the user, there are times where the movement is zero,whether because the user is firmly holding the camera or due to changeof movement direction the movement speed is zero at a certain instance.Such criteria may not need to be absolute. In addition such criteria maybe based on one or more 1-dimensional vectors as opposed to the full twodimensional image. In other words, the criteria 222 may be satisfied ifthe image is blurred horizontally, but no vertical movement is recordedand vice versa, due to the fact that the motion may be mathematicallydescribed in orthogonal vectors, thus separable. More straight forwardcriteria will be chronological, saving images every predefined timewhich can be equal or slower to the speed the preview images aregenerated. Other criteria may be defined such as related to theexposure, whether the preview reached focus, whether flash is beingused, etc.

Finally, the full resolution image acquired at 280 is saved, 282.

After the full resolution image is saved, 282, it is loaded into memory292 and the preview image or images are loaded into memory as well, 294.Together the preview and final images are the input of the process whichcalculates the PSF, 110.

A description of two different methods of calculating the PSF areillustrated in FIGS. 4 and 5.

FIG. 4 shows an embodiment 500 for extracting a PSF using a singlepreview image.

In this embodiment, the input is the finally acquired full resolutionimage 511, and a saved preview image 512. Prior to creating the PSF, thepreview and final image have to be aligned. The alignment can be aglobal operation, using the entire images, 511 and 512. However, the twoimages may not be exact for several reasons.

Due to the fact that the preview image and the final full resolutionimage differ temporally, there may not be a perfect alignment. In thiscase, local alignment, based on image features and using techniquesknown to those skilled in the art, will normally be sufficient. Theprocess of alignment may be performed on selected extracted regions 520,or as a local operation. Moreover, this alignment is only required inthe neighborhood of the selected region(s) or feature(s) used for thecreation of the PSF. In this case, matching regions of the fullresolution and preview image are extracted, 521 and 522. The process ofextraction of such regions may be as simple as separating the image intoa grid, which can be the entire image, or fine resolution regions. Othermore advanced schemes will include the detection of distinct regions ofinterest based on a classification process, such as detecting regionswith high contrast in color or exposure, sharp edges or otherdistinctive classifiers that will assist in isolating the PSF. Onefamiliar in the art is aware of many algorithms for analyzing anddetermining local features or regions of high contrast; frequencytransform and edge detection techniques are two specific examples thatmay be employed for this step, which may further include segmentation,feature extraction and classification steps.

The preview image 512 is normally, but not necessarily, of lowerresolution than the full resolution image 511, typically being generatedby clocking out a subset of the sensor cells or by averaging the rawsensor data. Therefore, the two images, or alternatively the selectedregions in the images, need to be matched in pixel resolution, 530. Inthe present context “pixel resolution” means the size of the image, orrelevant region, in terms of the number of pixels constituting the imageor region concerned. Such a process may be done by either upsampling thepreview image, 532, downsampling the acquired image, 531, or acombination thereof. Those familiar in the art will be aware of severaltechniques best used for such sampling methods.

Now we recall from before that:

-   -   A two dimensional image I is given as I(x,y).    -   A motion point spread function describing the blurring of image        I is given as MPSF(I).    -   The degraded image I′(x,y) can be mathematically defined as the        convolution of I(X,Y) and MPSF(x,y) or        I′(x,y)=I(x,y){circle around (×)}MPSF(x,y)   (Eq. 1)

Now it is well known that where a mathematical function, such as theaforementioned MPSF(x,y), is convoluted with a Dirac delta functionδ(x,y) that the original function is preserved. Thus, if within apreview image a sharp point against a homogenous background can bedetermined, it is equivalent to a local occurrence of a 2D Dirac deltafunction within the unblurred preview image. If this can now be matchedand aligned locally with the main, blurred image I′(x,y) then thedistortion pattern around this sharp point will be a very closeapproximation to the exact PSF which caused the blurring of the originalimage I(x,y). Thus, upon performing the alignment and resolutionmatching between preview and main images the distortion patternssurrounding distinct points or high contrast image features, are, ineffect, representations of the 2D PSF, for points and representation ofa single dimension of the PSF for sharp, unidirectional lines.

The PSF may be created by combining multiple regions. In the simplecase, a distinguished singular point on the preview image and itscorresponding motion blurred form of this point which is found in themain full-resolution image is the PSF.

However, as it may not always be possible to determine, match and align,a single distinct point in both preview and full resolution image, it isalternatively possible to create a PSF from a combination of theorthogonal parts of more complex features such as edges and lines.Extrapolation to multiple 1-D edges and corners should be clear for onefamiliar in the art. In this case multiple line-spread-functions,depicting the blur of orthogonal lines need to be combined and analysedmathematically in order to determine a single-point PSF.

Due to statistical variances this process may not be exact enough todistinguish the PSF based on a single region. Therefore, depending onthe processing power and required accuracy of the PSF, the step offinding the PSF may include some statistical pattern matching orstatistical combination of results from multiple regions within an imageto create higher pixel and potentially sub pixel accuracy for the PSF.

As explained above, the PSF may not be shift invariant. Therefore, theprocess of determining the right PSF may be performed in various regionsof the image, to determine the variability of the PSF as a function oflocation within the image.

FIG. 5 shows a method 600 of extrapolating a PSF using multiple previewimages.

In this embodiment, the movement of the image is extrapolated based onthe movement of the preview images. According to FIG. 5, the input forthis stage is multiple captured preview images 610, and the fullresolution image 620. All images are recorded with an exact time stampassociated with them to ensure the correct tracking. In most cases,preview images will be equally separated, in a manner of several imagesper second. However, this is not a requirement for this embodiment aslong as the interval between images, including the final full resolutionimage, is known.

One or more distinctive regions in a preview image are selected, 630. Bydistinctive, one refers to a region that can be isolated from thebackground, such as regions with noticeable difference in contrast orbrightness. Techniques for identifying such regions are well known inthe art and may include segmentation, feature extraction andclassification.

Each region is next matched with the corresponding region in eachpreview image, 632. In some cases not all regions may be accuratelydetermined on all preview images, due to motion blurring or objectobscurations, or the fact that they have moved outside the field of thepreview image. The coordinates of each region is recorded, 634, for thepreview images and, 636, for the final image.

Knowing the time intervals of the preview images, one can extrapolatethe movement of the camera as a function of time. When the fullresolution image 620 is acquired, the parameter that needs to berecorded is the time interval between the last captured preview imageand the full resolution image, as well as the duration of the exposureof the full resolution image. Based on the tracking before the image wascaptured, 634, and the interval before and duration of the final image,the movement of single points or high contrast image features can beextrapolated, 640, to determine the detailed motion path of the camera.

This process is illustrated in FIG. 6. According to this figure multiplepreview images 902, 904, 906, 908 are captured. In each of them aspecific region 912, 914, 916, 918 is isolated which corresponds to thesame feature in each image. The full resolution image is 910, and in itthe region corresponding to 912, 914, 916, 918 is marked as 920. Notethat 920 may be distorted due to motion blurring.

Tracking one dimension as a function of time, the same regions areillustrated in 930 where the regions are plotted based on theirdisplacement 932, as a function of time interval 932. The objects 942,944, 946 948 and 950 correspond to the regions 912, 914, 916, 918 and920.

The motion is calculated as the line 960. This can be done usingstatistical interpolation, spline or other curve interpolation based ondiscrete sampling points. For the final image, due to the fact that thecurve may not be possible to calculate, it may also be done viaextrapolation of the original curve, 960.

The region of the final acquired image is enlarged 970 for betterviewing. In this plot, the blurred object 950 is depicted as 952, andthe portion of the curve 690 is shown as 962. The time interval in thiscase, 935 is limited to the exact length in which the exposure is beingtaken, and the horizontal displacement 933, is the exact horizontalblur. Based on that, the interpolated curve, 952, within the exposuretime interval 935, produces an extrapolation of the motion path 990.

Now an extrapolation of the motion path may often be sufficient to yielda useful estimate of the PSF if the motion during the timeframe of theprinciple acquired image can be shown to have practically constantvelocity and practically zero acceleration. As many cameras nowincorporate sensitive gyroscopic sensors it may be feasible to determinesuch information and verify that a simple motion path analysis isadequate to estimate the motion blur PSF.

However when this is not the case (or where it is not possible toreliably make such a determination) it is still possible to estimate thedetailed motion blur PSF from a knowledge of the time separation andduration of preview images and a knowledge of the motion path of thecamera lens across an image scene. This process is illustrated in FIGS.7-a and 7-b and will now be described in more detail.

Any PSF is an energy distribution function which can be represented by aconvolution kernel k(x,y)→w where (x,y) is a location and w is theenergy level at that location. The kernel k must satisfy the followingenergy conservation constraint:∫∫k(x,y)dx dy=1,which states that energy is neither lost nor gained by the blurringoperation. In order to define additional constraints that apply tomotion blur PSFs we use a time parameterization of the PSF as a pathfunction, f(t)→(x,y) and an energy function h(t)→w. Note that due tophysical speed and acceleration constraints, f(t)should be continuousand at least twice differentiable, where f′(t) is the velocity of the(preview) image frame and f″(t) is the acceleration at time t. By makingthe assumption that the scene radiance does not change during imageacquisition, we get the${{\int_{t}^{t + {\delta\quad t}}{{h(t)}\quad{\mathbb{d}t}}} = \frac{\delta\quad t}{t_{end} - t_{start}}},{{\delta\quad t} > 0},{t_{start} \leq t \leq {t_{end} - {\delta\quad t}}},$additional constraint:

-   -   where [t_(start), t_(end)] is the acquisition interval for a        (preview) image. This constraint states that the amount of        energy which is integrated at any time interval is proportional        to the length of the interval.

Given these constraints we can estimate a continuous motion blur PSFfrom discrete motion samples as illustrated in FIGS. 7-a and 7-b. Firstwe estimate the motion path, f(t), by spine interpolation as previouslydescribed above and as illustrated in FIG. 6. This path [1005] isfurther illustrated in FIG. 7-a.

Now in order to estimate the energy function h(t)along this path we needto determine the extent of each image frame along this interpolatedpath. This may be achieved using the motion centroid assumptiondescribed in Ben-Ezra et al and splitting the path into frames with a1-D Voronoi tessellation as shown in FIG. 7-a. Since the assumption ofconstant radiance implies that frames with equal exposure times willintegrate equal amounts of energy, we can compute h(t) for each frame asshown in FIG. 7-b. Note that as each preview frame will typically havethe same exposure time thus each rectangle in FIG. 7-b, apart from themain image acquisition rectangle will have equal areas. The area of themain image rectangle, associated with capture frame 5 [1020] in thisexample, will typically be several time larger than preview image framesand may be significantly more than an order of magnitude larger if theexposure time of the main image is long.

The resulting PSF determined by this process is illustrated in FIG. 7-band may be divided into several distinct parts. Firstly there is the PSFwhich is interpolated between the preview image frames [1052] and shownas a solid line; secondly there is the PSF interpolated between the lastpreview image and the midpoint of the main acquired image [1054];thirdly there is the extrapolation of the PSF beyond the midpoint of themain acquired image [1055] which, for a main image with a long exposuretime—and thus more susceptible to blurring—is more likely to deviatefrom the true PSF. Thus it may be desirable to acquire additionalpostview images, which are essentially images acquired through the samein-camera mechanism as preview images except that they are acquiredafter the main image has been acquired. This technique will allow afurther interpolation of the main image PSF [1056] with the PSFdetermined from at least one postview image.

The process may not be exact enough to distinguish the PSF based on asingle region. Therefore, depending on the processing power and accuracyneed, the step of finding the PSF may include some statistical patternmatching of multiple regions, determining multiple motion paths, thuscreating higher pixel and potentially sub pixel accuracy for the PSF.

Advantageously, a determination may be made whether a threshold amountof camera motion blur has occurred during the capture of a digitalimage. The determination is made based on a comparison of a least twoimages acquired during or proximate to the exposure period of thecaptured image. The processing occurs so rapidly, either in the cameraor in an external processing device, that the image blur determinationoccurs in “real time”. The photographer may be informed and/or a newimage capture can take place on the spot due to this real time imageblur determination feature. Preferably, the determination is made basedon a calculated camera motion blur function, and further preferably, theimage may be de-blurred based on the motion blur function, eitherin-camera or in an external processing device in real time or later on.In one embodiment, a same mechanism that determines auto focus (e.g.,local contrast gradient or edge detection) is used for motionevaluation. In particular, the process of auto focusing is done in realtime and therefore the mechanism is fast. Such mechanism as understoodby those skilled in the art, may be used in the qualitative andquantitative determination of motion blur.

While an exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention as set forth in the appended claims and structural andfunctional equivalents thereof.

In addition, in methods that may be performed according to preferredembodiments herein and that may have been described above, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations, except for those where a particularorder may be expressly set forth or where those of ordinary skill in theart may deem a particular order to be necessary.

In addition, all references cited herein as well as the background,invention summary, abstract and brief description of the drawings areincorporated by reference into the description of the preferredembodiment as disclosing alternative embodiments.

1. A digital image acquisition system, comprising an apparatus forcapturing digital images and a digital processing component fornotifying the photographer upon determining that a threshold cameramotion blur has occurred in a captured digital image based on acomparison of at least two sets of image data each acquired within atemporal range that includes an exposure period of the captured digitalimage, and times proximately before and after said exposure period, andof nominally the same scene as that of the captured digital image.
 2. Asystem according to claim 1, wherein the determining that a thresholdcamera motion blur has occurred is based on determining a motion blurfunction.
 3. A system according to claim 2 wherein the camera motionblur function is mathematically defined as a point spread function(PSF).
 4. A system according to claim 2, wherein said digital processingcomponent further de-blurs the captured image using said determinedcamera motion blur function.
 5. A system according to claim 1, whereinat least one of the sets of image data that are compared comprises apreview image.
 6. A system according to claim 1, wherein a first set ofimage data is compared to a second set of image data comprising a lowerpixel resolution than the first set and wherein said digital processingcomponent is arranged to match the pixel resolutions of the first andsecond image data sets.
 7. A system according to claim 6, wherein saiddigital processing component is arranged to match the pixel resolutionsof the first and second image data sets by at least one of up-samplingthe first image data set and sub-sampling the second image data set. 8.A system according to claim 1, wherein said digital processing componentis arranged to align at least portions of the at least two image datasets that are compared.
 9. A system according to claim 8, wherein thealignment is a global alignment on entire or substantially entireimages.
 10. A system according to claim 1, wherein said digitalprocessing component uses the captured image and a single referenceimage as the image data sets that are compared, and determines a motionblur function based on the comparison.
 11. A system according to claim10, wherein said digital processing component identifies at least onecharacteristic of the reference image which is relatively less blurredthan the corresponding characteristic of the captured image, andcalculates said blur function in respect of said characteristic.
 12. Asystem according to claim 11, wherein said digital processing componentde-blurs the captured image by de-convolving the captured image usingthe blur function.
 13. A system according to claim 1, wherein saiddigital processing component uses the captured image and multiplereference images taken at successive points in time.
 14. A systemaccording to claim 13, wherein said digital processing componentcalculates a trajectory of at least one characteristic in the referenceimages, extrapolates such characteristic on to the captured image, andcalculates a motion blur function in respect of said characteristic. 15.A system according to claim 14, wherein said digital processingcomponent de-blurs the captured image by de-convolving the capturedimage using the blur function.
 16. A system according to claim 1,wherein said digital processing component is configured to visuallyinform the photographer if it determines that the captured image isblurred.
 17. A system according to claim 1, wherein said digitalprocessing component is configured to inform the photographer with anaudio signal if it determines that the captured image is blurred.
 18. Asystem of according to claim 1, wherein the digital image capturingapparatus is signalled to capture another image if the digitalprocessing component determines that the captured image is blurred. 19.A system according to claim 1, wherein said digital image acquisitionsystem comprises a portable digital camera.
 20. A system as claimed inclaim 19, wherein said digital processing component is located in saidportable digital camera.
 21. A system according to claim 1, wherein saiddigital image acquisition system comprises a combination of a portabledigital camera and an external processing device.
 22. A system asclaimed in claim 20, wherein said digital processing component islocated in said external processing device.
 23. A system as in claim 1,wherein said at least two image data sets comprise said captured imageand a reference image taken outside the exposure period of said capturedimage.
 24. A system as in claim 23, wherein said reference image istaken before said exposure period of said captured image.
 25. A systemaccording to claim 23 wherein the digital processing componentdetermines a camera motion blur function based on the comparison,wherein the camera motion blur function is mathematically defined as apoint spread function (PSF).
 26. A system according to claim 25, whereinsaid digital processing component further de-blurs the captured imageusing said determined camera motion blur function.
 27. A systemaccording to claim 25, wherein the reference image comprises a previewimage.
 28. A system according to claim 23, wherein the reference imagecomprises a lower pixel resolution than the captured image, and whereinsaid digital processing component is arranged to match the pixelresolutions of the captured image and the reference image.
 29. A systemaccording to claim 28, wherein said digital processing component isarranged to match the pixel resolutions of the captured image and thereference mage by at least one of up-sampling the reference image andsub-sampling the captured image.
 30. A system according to claim 23,wherein said digital processing component is arranged to align at leastportions of the captured image and the reference image.
 31. A systemaccording to claim 23, wherein the motion blur determining is performedin real time relative to the capturing of the digital image.
 32. Asystem according to claim 1, wherein said apparatus comprises an imagesensor component and said at least two sets of image data are acquiredby said same image sensor component.
 33. A system according to claim 32,wherein said apparatus further comprises an optical system and said atleast two sets of image data are acquired along said same optical path.34. In a digital image acquisition system including an apparatus forcapturing digital images, a method comprising notifying the photographerupon determining that a threshold camera motion blur has occurred in acaptured digital image based on a comparison of at least two sets ofimage data each acquired within a temporal range that includes anexposure period of the captured digital image, and times proximatelybefore and after said exposure period, and of nominally the same sceneas that of the captured digital image.
 35. A method according to claim34, wherein said at least two images comprise said captured image and areference image taken outside the exposure period of said capturedimage.
 36. A method according to claim 35, wherein said reference imageis taken before said exposure period of said captured image.
 37. Amethod according to claim 35, wherein the determining comprises matchingpixel resolutions of the reference and captured images by at least oneof up-sampling a reference image and sub-sampling the captured image.38. A method according to claim 35, wherein the determining comprisesaligning at least portions of the captured image and a reference image.39. A method according to claim 35, further comprising identifying atleast one characteristic of the reference image which is relatively lessblurred than the corresponding characteristic of the captured image, andcalculating a motion blur function in respect of said characteristic.40. A method according to claim 35, further comprising calculating atrajectory of at least one characteristic in the reference image,extrapolating such characteristic onto the captured image, andcalculating a motion blur function in respect of said characteristic.41. A method according to claim 34, further comprising determining acamera motion blur function based on said comparison, and wherein thecamera motion blur function comprises a point spread function (PSF). 42.A method according to claim 41, further including de-blurring thecaptured image using said determined camera motion blur function.
 43. Amethod according to claim 42, wherein the de-blurring comprisesde-convolving the captured image using the blur function.
 44. A methodaccording to claim 34, further including visually informing thephotographer when the captured image is determined to be blurred.
 45. Amethod according to claim 34, further including informing thephotographer by audio signal when the captured image is determined to beblurred.
 46. A method according to claim 34, further includingtransferring the at least two images from the digital image capturingapparatus to an external processing device for determining a motion blurfunction.
 47. A method according to claim 34, wherein the motion blurdetermining is performed in real time relative to the capturing of thedigital image.
 48. A method according to claim 34, further comprisingacquiring images corresponding to said at least two image data sets witha same image sensor component.
 49. A method according to claim 47,wherein said capturing of said images occurs along a same optical path.50. One or more storage devices having digital code embedded therein forprogramming one or more processors to perform a camera motion blurdetermining method that comprises capturing digital images, andnotifying the photographer upon determining that a threshold cameramotion blur has occurred in a captured digital image based on acomparison of at least two sets of image data each acquired within atemporal range that includes an exposure period of the captured digitalimage, and times proximately before and after said exposure period, andof nominally the same scene as that of the captured digital image. 51.The one or more storage devices of claim 50, wherein said at least twoimages comprise said captured image and a reference image taken outsidethe exposure period of said captured image.
 52. The one or more storagedevices of claim 51, wherein said reference image is taken before saidexposure period of said captured image.
 53. The one or more storagesdevices of claim 51, wherein the determining comprises matching pixelresolutions of the reference and captured images by at least one ofup-sampling a reference image and sub-sampling the captured image. 54.The one or more storage devices of claim 51, wherein the determiningcomprises aligning at least portions of the captured image and areference image.
 55. The one or more storage devices of claim 51, themethod further comprising identifying at least one characteristic of thereference image which is relatively less blurred than the correspondingcharacteristic of the captured image, and calculating a camera motionblur function in respect of said characteristic.
 56. The one or morestorage devices of claim 51, the method further comprising calculating atrajectory of at least one characteristic in the reference image,extrapolating such characteristic onto the captured image, andcalculating a camera motion blur function in respect of saidcharacteristic.
 57. The one or more storage devices of claim 50, themethod further comprising determining a camera motion blur function, andwherein the camera motion blur function comprises a point spreadfunction (PSF).
 58. The one or more storage devices of claim 57, furtherincluding de-blurring the captured image using said determined cameramotion blur function.
 59. The one or more storage devices of claim 58,wherein the de-blurring comprises de-convolving the captured image usingthe blur function.
 60. The one or more storage devices of claim 50, themethod further including visually informing the photographer when thecaptured image is determined to be blurred.
 61. The one or more storagedevices of claim 50, the method further including informing thephotographer by audio signal when the captured image is determined to beblurred.
 62. The one or more storage devices of claim 50, the methodfurther including transferring the at least two images from the digitalimage capturing apparatus to an external processing device fordetermining a motion blur function.
 63. The one or more storage devicesof claim 50, wherein the motion blur determining is performed in realtime relative to the capturing of the digital image.
 64. The one or morestorage devices of claim 50, the method further comprising acquiringimages corresponding to said at least two image data sets with a sameimage sensor component.
 65. The one or more storage devices of claim 64,wherein said capturing of said images occurs along a same optical path.66. A digital image acquisition system, comprising an apparatus forcapturing digital images and a digital processing component fornotifying the photographer upon determining that a threshold cameramotion blur has occurred in a captured digital image based on a realtime analysis of the captured digital image.
 67. A system according toclaim 66, wherein the determining that a threshold camera motion blurhas occurred is based on qualitatively determining a motion blur.
 68. Asystem according to claim 67, wherein the camera motion blur ismathematically defined as a point spread function (PSF).
 69. A systemaccording to claim 67, wherein said digital processing component furtherde-blurs the captured image using said determined camera motion blurfunction.
 70. A system according to claim 67, wherein said qualitativelydetermining a motion blur is performed using an auto-focus mechanism ofthe camera as the digital processing component.
 71. A system accordingto claim 66, wherein said digital processing component is configured toinform the photographer if it determines that the captured image isblurred.
 72. A system of according to claim 71, wherein the digitalimage capturing apparatus is signalled to capture another image if thedigital processing component determines that the captured image isblurred.
 73. A system of according to claim 66, wherein the digitalimage capturing apparatus is signalled to capture another image if thedigital processing component determines that the captured image isblurred.
 74. A system according to claim 66, wherein said digital imageacquisition system comprises a portable digital camera.
 75. A systemaccording to claim 74, wherein said digital processing component islocated in said portable digital camera.
 76. A system according to claim66, wherein said determining that a threshold camera motion blur hasoccurred in the captured digital image is further based on a comparisonof at least two sets of image data each acquired within a temporal rangethat includes an exposure period of the captured digital image, andtimes proximately before and after said exposure period, and ofnominally the same scene as that of the captured digital image.